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[XTKOIK’CTION 

The  l idcrai  COBOL  Compiler  Testing  Service  (l'CCTS) 
i'  an  activity  of  the  Software  Development  Division  of  tiie 
Department  of  the  Navy,  Automatic  Data  Processing 
Equipment  Selection  Office  (ADPESO).  Since  July  1,  1972, 
all  COBOL  compilers  brought  into  the  Federal  Government 
have  to  be  identified  as  implementing  one  of  the  levels  of 
the  Federal  COBOL  Standard.  The  National  Bureau  of 
Standards,  which  has  the  responsibility  for  the  development 
and  maintenance  of  Federal  ADP  Standards,  has  delegated 
to  the  Department  of  Defense,  and  thereby  to  ADPESO, 
the  responsibility  for  the  operation  of  a Government-aide 
COBOL  Compiler  Testing  Service.  This  responsibility  is 
di.s  barged  by  the  l'CCTS  through  the  implementation  and 
maintenance  of  the  COBOL  Compiler  Validation  System,1 
a comprehensive  set  of  routines  used  to  test  COBOL  com- 
pilers for  compliance  with  the  Federal  COBOL  Standard  as 
prescribed  in  Federal  Information  Processing  Standards 
Publication  _’l  til! ’S  PER  21),*  published  by  the  National 
Bureau  of  Standards 

This  paper  addresses  several  questions  that  arise  in  com- 
piler validation.  Why  validate  compilers  for  conformance 
to  a standard?  How  is  the  validation  performed?  What 
experiences  have  been  gained,  and  what  conclusions  ran  be 
derived  from  them?  The  questions  will  be  discussed  in  turn. 

WHY  VALIDATE  COMPILERS’ 

The  mrposi  of  validating  a compiler  is  to  ensure  that 
syntactically  correct  programs  compile  and  execute  without 
abnormal  termination,  and  that  the  semantics  of  the  lan- 
guago  being  translat'd  are  correctly  interpreter!.  Also, 

>'■  r.  appropriate  to  the  language,,  validation  should 
(Hunt  out  the  impact  of  implementor  defined  specifications 
which  are  allowed  bv  the  standard. 

There  are  three  phases  in  the  computer  systems  acquisition 
cycle  during  which  a validation  is  important.  Prior  to  selec- 
tion, a validation  of  the  compilers  for  the  various  systems 
being  promised  may  constitute  a part  of  the  systems  evalua- 
tion After  a computer  system  has  been  selected,  a valida- 
tion of  'In-  present  compiler  and  a validation  ol  the  compiler 
* ■ b"  procur'd  disclose  the  areas  of  nonconformance  in  both 


compilers.  The  effort  required  in  converting  existing  pro- 
grams to  the  new  system  can  then  be  realistically  estimated 
prior  to  the  changeover.  After  the  delivery  of  a new  com- 
puter system,  but  prior  to  acceptance,  a compiler  validation 
will  reveal  areas  where  a compiler  does  not  meet  the  terms 
of  t he  contract 

Our  experience  with  the  Validation  System  has  shown 
that  continuing  benefits  accrue  from  being  aware  of  a com- 
piler’s status  vis-a-vis  its  language  standard  Compiler 
validation  for  computer  systems  which  have  already  been 
acquired  and  are  in  present  use  can  serve  to  point  out  which 
language  elements  do  not  operate  correctly  and  therefore 
should  not  be  used.  A validation  is  particularly  useful  wl.-  n 
a new  version  of  a compiler  or  operating  system  is  released, 
since  it  will  immediately  reveal  errors  in  the  revised  software. 

If  a user  lias  access  to  several  different  computer  systems 
and  is  doing  program  development  on  all  of  these,  lie  must 
know  what  language  elements  conform  to  the  standard  on 
each  of  the  systems.  Validation  of  the  compilers  on  each 
system  shows  which  language  elements  perform  correctly. 
By  w riting  programs  ’ using  only  these  elements.  ‘ a ' no' 
ensures  program  portability. 

SCOPE  OF  VALIDATION 

Errors  in  compiling  a program  may  arise  from  a single 
statement  or  a particular  sequence  of  statements  Since 
validation  verities  that  individual  language  elements  are 
processed  correctly,  errors  in  combining  language  elements 
may  exist  even  though  each  of  iho  separate  elements  are 
processed  correctly 

A validation  is  not  concerned  with  the  efficiency  of  the 
object  code  generated,  but  only  tests  if  the  code  is  produced 
correctly.  A validation  system  does  not  test  implementor 
extensions  to  the  language.  If  the  implementor  ixtensions 
cause  problems  in  the  standard  language  elements,  a valida- 
tion will  identify  these  errors,  but  any  errors  in  the  use  of 
the  language  extensions  themselves  will  not  be  discovered 
during  validation. 

Finally,  while  a validation  identifies  problem  areas  in  tin 
use  of  standard  !anguag<  elements  with  a given  compiler,  it 
cannot  indicate  the  rami  heat  ions  of  the  compiler  errors 
discovered 
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FCCTS  COBOL  COMPILER  VALIDATION 

The  validation  of  COBOL  compilers  by  the  Federal 
COBOL  Compiler  Testing  Service  is  performed  using  the 
COBOL  Compiler  Validation  System  (CCVS/,  which  was 
developed  by  the  Department  of  the  Navy  Programming 
Languages  Section  under  the  direction  of  Capt.  Grace  M. 
Hopper.  USNU.  The  CCVS  consists  of  audit  routines,  their 
related  data  and  an  executive  routine.  Each  audit  routine  is 
a COBOL  program,  and  includes  many  tests  of  individual 
language  elements.  Supporting  procedures  indicating  the 
results  of  the  tests  are  included  in  each  routine.  The  audit 
routines  of  the  CCVS  collectively  contain  the  features  of 
Federal  Standard  COBOL. 

There  are  certain  adjustments  which  must  be  made 
before  the  audit  routines  can  be  compiled  upon  a given 
computer  system.  First,  names  allowed  by  the  COBOL 
standard  to  be  implementor  defined  must  be  inserted  into 
the  audit  routines  before  they  can  be  compiled.  Second, 
system  control  cards  are  required  in  order  to  compile  and 
execute  a COBOL  program.  File  specification  and  allocation 
are  also  regulated  by  system  control  cards,  and  additional 
control  cards  are  usually  required  by  programs  using  the 
COBOL  SORT  verb.  Third,  the  given  system  configuration 
may  not  include  a hardware  device  or  capability'  required 
by  some  of  the  test  procedures  in  the  CCVS,  for  example  a 
system  which  does  not  support  multiple  unit  assignment 
fc  r mass  storage  devices.  All  references  to  multiple  units 
must  be  deleted  for  the  proper  compilation  of  the  audit 
routines  on  that  system.  

EXECUTING  THE  AUDIT  ROUTINES 

Input  parameters  to  the  CCVS  executive  routine’  specify 
the  implementor  names,  hardware  dependent  language 
elements  to  be  deleted,  and  the  operating  system  control 
cards  required  to  compile  the  audit  routines  on  a given 
computer  system.  The  executive  routine  creates  a file  con- 
taining the  audit  routines  with  implementor  names  inserted 
in  the  proper  place  in  the  source  code,  and  the  operating 
system  control  cards  required  for  compiling  and  executing 
each  routine. 

The  audit  routines  in  the  CCVS  consist  of  source  code 
which  is  syntactically  correct;  the  routines  do  not  contain 
any  tests  which  deliberately  introduce  incorrect  syntax. 
Thus,  each  audit  routine  is  expected  to  compile  without 
errors.  (This  is  frequently  not  the  case.  We  have  encoun- 
tered "Standard''  compilers  where  syntactically  correct 
source  code  causes  fatal  diagnostic  messages,  compiler 
aborts,  and  even  compiler  loops.) 

When  an  audit  routine1  does  not  compile,  or  complete 
execution  normally,  the  sourre  code  containing  the  language 
elements  which  the  compiler  could  not  handle  is  modified 
or  deleted  The  tests  in  the  PROCEDURE  DIVISION  of 
the  audit  routines  are  coded  so  that  a test  is  deleted  by 
inserting  NOTE  at  the  beginning  of  the  paragraph  contain- 
ing the  test.  This  results  in  the  entire  paragraph  being 


treated  as  a comment.  The  source  paragraph  following  the 
deleted  paragraph  contains  procedures  which  indicate  the 
test  has  been  deleted. 

The  CCVS  executive  routine  contains  an  editing  capability 
which  permits  addition,  deletion,  or  replacement  of  source 
lines  in  the  audit  routines.  After  an  audit  routine  has  been 
modified  so  that  it  consists  of  only  the  language  elements 
that  the  compiler  accepts,  the  routines  are  again  compiled 
and  executid. 

-Vll  of  the  supporting  procedures  for  verifying  whether  a 
test  passes  or  fails  is  contained  in  each  routine.  An  output 
report  is  produced  indicating  the  actual  results  of  each  tost, 
and,  when  a test  fails,  the  expected  result. 

ADDITIONAL  INFORMATION  FROM  COMPILER 
VALIDATION 

A compiler  validation  identifies  many  characteristics 
which  can  be  used  in  comparing  compilers.  Due  to  compiler 
errors,  valid  syntax  in  the  audit  routines  may  be  rejected 
or  the  resultant  object  program  may  abort  during  exe  cu- 
tion. The  running  of  the  CCVS  on  a system  supplies  informa- 
tion concerning  the  effectiveness  of  diagnostic  messages 
The  effort  required  in  locating  the  source  code  which  caused 
execution  to  terminate  abnormally  can  also  be  assessid. 

The  procedures  used  in  validating  a compiler  give  infor- 
mation on  a system's  ability  to  execute  a program  after 
fatal  compilation  errors  have  been  diagnosed,  or  the  effec- 
tiveness of  a system  supplied  option  for  skipping  execution 
if  fatal  errors  are  encountered 

Some  minor  programming  aids  may  also  be  discovered 
during  a validation.  The  audit  routines  can  be  compiled 
using  options  for  cross  reference  listings  of  data-names  and 
procedure-names.  Some  compilers  flag  blank  cards.  In  some 
cast's,  we  have  uncovered  hindrances.  The  suppression  of 
the  printing  of  the  contents  of  columns  78  through  80  on  a 
source  listing  is,  for  example,  a hindrance  to  anyone  run- 
ning the  CCVS,  since  the  CCVS  executive  routine  uses 
columns  78-80  to  indicate  whether  a source  line  has  been 
replaced  or  added. 

THE  VALIDATION  SUMMARY  REPORT 

The  output  of  a validation  is  a set  of  listings  from  the 
executive  routine  documenting  the  steps  taken  in  preparing 
programs  jobs  for  execution;  the  compilation  of  each  pro- 
gram; and  the  execution  report  of  each  program.  These 
listings  constitute  the  raw  data  from  which  a Validation 
Summary  Report  (VSR)  is  produced.  (Any  attempt  to  use 
the  raw  results  for  evaluating  a compiler  would  be  painful 
indeed  due  to  the  volume  of  paper  involvid.)  The  VSR 
provides  the  following  information: 

(a)  The  status  of  the  compiler  in  relation  to  each  of  the 
four  Federal  levels  of  COBOL  as  defined  in  FIPS 
PUB  21. 

(b)  A list  of  language  elements  whose  implementation 
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is  not  consistent  with  the  language  specification 
(American  National  Standard  COBOL  X3.23-19fi8).* 

(c)  A list  of  language  elements  which  are  not  imple- 
mented due  to  a lack  of  hardware'  ne  cessary  to  sup- 
port those  elements  (e.g.,  read  reversed,  hardware 
switches,  etc.).  The  language  specification  does  not 
require  the  implementation  of  certain  elements  if 
they  are  dependent  on  specific  hardware  devices, 
and  the  system  supporting  the  COBOL  compiler 
itself  does  not  support  that  device. 

(d)  Information-only  items.  These  are  necessitated  due 
to  the  existence  of  imprecise'  language  specifications 
in  the'  Standard. 

(e)  Compiler  characteristics  noticed  by  the  validation 
team.  This  includes  the  usefulness  of  diagnostic 
messages  issued,  format  of  the  source  preigrarn 
listing,  and  timings  and  memory  requirements  for 
the  compiler  and  individual  audit  routine's. 

COBOL  COMPILER  PROBLEMS  DISCOVERED 
DURING  VALIDATIONS 

The  FCCTS  has,  since  its  establishment,  validated  com- 
pile  rs  supplied  by  many  different  vendors.  Many  problem 
areas  have'  be'en  unceivered  during  these'  validations.  Some 
of  the  problems  are  common  to  many  compilers;  othi'rs 
eiccurred  emly  in  particular  ones. 

In  this  se'etion  we  pre'sent  seeme  of  the  common  preiblem 
areas -we  have  discovered-  in  “Standard”  COBOL  compilers. 
The  preiblems  are  grouped  by  the-  COBOL  functional  pro- 
cessing module  in  which  they  are  found. 

Nucleus 

In  a NOTE  character  string,  any  combination  of  charac- 
ters front  the  computer's  character  set  is  treated  as  a com- 
ment: the-  string  appears  on  the  source  listing,  but  is  not 
compiled.  Frenn  the'  NOTE  tests  included  in  the  CCVS  we 
have  founel  that  most  COBOL  compilers  che-ck  the  syntax 
etf  a NOTE  statement  As  an  example,  une  of  the  tests  is  a 
NOTE  sentence'  containing  a single  "(QUOTE)  character. 
Most  compilers  generate-  a message  indicating  an  ille>gal 
alphanumeric  literal  format  was  used  since  an  alphanumeric 
literal  is  a character  string  ene'losed  in  quotes  If  the  NOTE 
statement  is  not  ,he  first  sentence  of  a paragraph,  the*  NOTE 
comment  ends  with  the*  first  period  followed  by  a space.  On 
many  systems  theiugh.  the  first  period,  with  or  without  a 
trailing  spae-e,  terminates  the  NOTE  ceimment  and  attempts 
are  made  to  compile  the"  rest  of  the  comment. 

There  are  tests  of  the  ADD  and  SUBTRACT  CORRE- 
SPONDING statements  with  matching  items  requiring 
five  le'vels  of  qualification.  Most  compilers  give  diagneistic 
error  message's  for  these  tests,  but  in  some  case's  the  com- 
pilation of  the  preigrarn  was  terminated. 

A te’st  of  the-  floating  insertion  editing  capability  included 
in  a Nucleus  module  auelit  reiutine  move's  000123.45  to  an 
elementary  item  with  a PICTURE  clause  WB999.99. 


The'  language'  specificatiein  states  that  any  of  the'  simple 
insertion  characters  (comma,  blank  and  zero)  immediately 
to  the:  right  of  floating  insertion  characters  are  part  of  the' 
fleiating  character-string  Thus,  the  contents  of  the  edited 
item  after  the-  tot  should  bei  $123.45.  The  result  usually 
obtained  for  this  te'st  is  $1)123.45. 

Sequential  anil  random  arccs.t 

The  size  of  the  data  reeords  for  a file  may  be  specified 
by  the1  RECORD  CONTAINS  integer-1  TO  inte‘ge‘r-2 
CHARACTERS  edauseo  The  size*  of  each  record  in  the  file 
is  de'fined  by  the'  individual  re-cord  descriptions  and  the 
RECORD  CONTAINS  clause  is  optional.  If  the-  clause  is 
used,  there  sheiuld  not  be  any  restrictions  in  the  File  1 In- 
scription. We  have  found  compilers  which  require  the  num- 
ber of  characters  in  each  record  tei  be>  a special  item  at  tie 
beginning  eif  a receird  description  when  this  clause-  is  used 
This  is  a nonstandard  restriction. 

One  audit  routine'  for  beith  sequential  and  random  access 
file  prex- easing  includes  a CLOSE  file  WITH  LOCK  state- 
ment. Attempts  are  then  made'  to  OPEN  and  READ  the 
kicked  file  during  the'  current  run  unit.  A file  that  has  be'en 
cleised  with  lock  cannot  be'  opened  again  during  execution 
eif  the  object  program.  Most  of  the  systems  te'sted  abort 
the  preigrarn  execution  with  an  error  message'  stating  that 
an  attempt  to  access  a locked  file  has  boon  made.  Some  of 
the  systems  return  data  and  continue  e-xecuting  as  if  the 
CLOSE  WITH  LOCK  had  not  been  encountered  -In  . me* 
case,  the*  program  went  into  an  execution  loop  whe-n  an 
attempt  to  read  a locked  file1  was  made. 

Table  handling 

In  the  COBOL  Table  Handling  Moduli',  the  OCCL'RS 
DEPENDING  ON  option  specifies  a table  whose  number 
of  occurrences  varie-s  during  e xecution.  The  number  of  items 
in  the  table  during  execution  depends  on  the  value  of  the 
data-name  in  the-  DEPENDING  ON  clause'.  If  a SEARCH 
statement  is  encountered  for  a variable  table',  the-  last 
entry  in  the  table  is  defined  by  the  contents  of  the'  date- 
name.  Some  compilers  accent  the  syntax  for  the  table  defini- 
tion, but  the  table  is  always  considered  te-  be  its  maximum 
size  in  a SEARCH  statenii'nt. 

Segmentation 

An  independent  program  segment  is  expe-cted  to  be  in 
its  initial  state  each  time  the-  se<gment  is  made  available  to 
the  program.  There  an-  compilers  which  do  not  restore-  the 
initial  state  of  independent  program  segments 

Library 

The  COPY  statement  with  the-  REPLACING  option 
allows  the  user  to  copy  library  fe-xt,  replacing  each  occur- 
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rcnco  of  a .vord  in  the  text  with  a new  word.  A variety  of 
problems  have  been  found  when  exercising  this  option  in 
the  audit  routines.  One  compiler  correctly  handled  u COPY 
REPI.AOINO  statement,  but  in  a subsequent  COPY 
without  the  REPLACING  option  of  the  same  library  text, 
words  were  replaced  which  should  not  have  been.  This 
caused  undefined  data-names  during  the  program  com- 
pilation. 

Main  compilers  place  restrictions  on  the  REPLACING 
option  which  are  not  in  the  Standard.  Some  of  these  restric- 
tions are  that  a qualified  name  could  not  be  replaced ; a 
data-name  could  not  be  replaced  by  a subscript  I'd  data- 
name;  or  a data-name  could  not  be  replaced  in  an  01  level 
entry  in  the  Data  Division. 

Sort 

A frequent  restriction  encountered  in  the  SORT  module 
is  a limitation  in  the  specification  of  the  minimum  number 
of  characters  in  a sort  record  description.  Usually  the  com- 
pilation of  the  cOBOL  source  program  will  not  cause  any 
diagnostic  messages,  but  when  the  sort  program  is  exe- 
cuted, a message  indicating  incorrect  record  length  is  pro- 
duced. 

Problems  have  arisen  when  one  of  the  sort  keys  is  defined 
as  a signed  numeric  field.  Tor  a sort  on  ascending  keys, 
negative  values  should  appear  before  positive  values.  In 
some  cases,  signed  numeric  items  have  not  been  sorted  in 
the  correct  order  because  comparison  was  based  on  the 
actual  binary  structure  of  the  data,  and  not  the  algebraic 
value  associated  with  the  data. 

Implementation  variations 

A file  may  contain  multiple  record  descriptions,  with  each 
record  having  a different  length.  It  is  important  for  a user 
to  know  how  much  external  storage  media  must  be  allocated 
for  records  of  a given  file.  In  order  to  compute  this  a user 
must  know  if  a given  implementation  always  writes  records 
of  the  maximum  length,  or  if  variable  length  records  are 
written.  This  would  be  especially  significant  if  most  of  the 
records  in  the  file  were  much  shorter  than  the  remainder. 
Though  not  required  by  the  standard,  a great  deal  of  ex- 
ternal storage  space  is  wasted  if  fixed  length  records  are 
written. 

There  are  procedures  in  both  sequential  and  random 
access  audit  routines  which  create  a file  containing  variable 
length  records,  and  in  the  subsequent  reading  of  the  file  test 
if  the  system  creates  all  records  as  the  maximum  fixed  length. 

As  a result  of  eur  validations,  we  have  changed  some  tests 
to  information-only  tests  because  of  language  ambiguities. 
For  example,  there  is  no  explicit  statement  in  the  Standard 
as  to  what  the  result  should  be  when  moving  a signed  nu- 
meric field  to  an  alphanumeric  field.  There  are  statements 
suggesting  that  any  appropriate  conversion  takes  place 
during  an  elementary  move,  but  there  could  be  doubt  as  to 


whether  the  elimination  of  a sign  is  included.  In  ord>  r to 
determine  the  result  for  a particular  compiler,  the  alpha 
numeric  item  is  tested  for  a numeric  value  after  moving  a 
+ 1 and  a —1  to  the  alphanumeric  item 
The  use  of  optional  words  is  only  for  readability  ami  they 
are  not  supposed  to  effect  the  meaning  of  the  statements 
However,  we  have  found  one  case  where  the  presence  ol 
the  optional  word  ‘IS’  changes  tin-  meaning  of  a statement 
One  of  the  audit  routine  tests  is 

IF  A = B AND  IS  NOT  GREATER  C OR  D 

The  presence  of  the  word  IS’  leaves  no  doubt  that  V T 
is  part  of  the  relational  operator  NOT  GREATER  V-  a 
result,  the  expansion  gives 

IF  A = B AND  A IS  NOT  GREATER  C OR 
.4  IS  SOT  UREA  TER  I). 

But  for  the  combined  abbreviated  relational  condition 
IF  A = B AND  NOT  GREATER  (’  OR  I), 

the.  language  specification  states  explicitly  that  the  word 
NOT  is  part  of  the  logical  connector  AND  NOT.  and  is 
not  part  of  the  relational  operator  NOT  GREATER  Tin- 
effect  of  the  rule  causes  the  statement  to  be  expanded  so 
that  the  NOT  is  associated  with  the  operand  called  and 
not  associated  with  the  operand  called  D 

IF  A = B AND  NOT  A GREATER  C OR  A UREA  TER  1). 

The  presence  of  the  word  IS’  in  the  statement  can  indeed 
violate  the  law  of  least  astonishment. 

The  discovery  of  problems  in  tests  such  as  those  de- 
scribed above  has  resulted  in  recommendations  for  language 
clarification  to  the  American  National  Standards  Institute 
Technical  Committee  X.i.14,  which  has  the  responsibility 
for  the  maintenance  of  X.}.23-i9ti8  COBOL. 

REASONS  FOR  FAILURES  IN  EXECUTING  THE 
AUDIT  ROUTINES 

There  are  si  vcral  reasons  why  a compiler  may  be  imple- 
mented in  such  a way  that  there  arc  discrepancies  between 
the  language  specifications  and  the  results  given  by  the 
compiler.  The  most  common  reason  for  discrepancies  is 
simply  due  to  logic  errors  in  the  compiler.  This  can  be 
attributed  to  lack  of  adequate  controls  in  producing  com- 
pilers. 

A second  reason  for  differences  in  the  implementation  is 
the  misinterpretation  of  the  language  specification  on  the 
part  of  the  implementor.  One  case  in  point  (which  has  been 
noticed  in  several  compilers)  is  the  use  of  the  word  THRU 
in  the  current  standard  (X3.23-1D68).  In  many  specifications 
throughout  the  document,  the  word  THRU  appears  as 
follows: 

PERFORM  prorodure-namo-I  THRU  procedure-name-2 
FILE-LIMIT  literal-1  THRU  literal-2,  etc. 

There  is  nothing  in  the  above  syntax  which  indicates  that 
the  words  THRU  and  THROUGH  are  interchangeable. 
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The  only  reference  that  establishes  this  little  known  fact 
is  the  reserved  word  list  where  they  are  shown  to  be  equiva- 
lent. This  probably  accounts  for  the  fact  that  we  have 
identified  several  compilers  which  do  not  allow  usage  of 
THROUGH. 

Another  problem  is  the  ambiguity  that  is  inherent  in  a 
language  as  complex  as  COBOL.  These  are  areas  in  the 
current  language  specification  that,  at  best,  are  ill-defined, 
and  the  implementor  must  make  a unilateral  decision  as  to 
tlie  direction  the  implementation  will  take.  A good  example 
would  be  the  default  action  the  compiler  takes  for  a WRITE 
statement  to  the  printer,  when  the  ADVANCING  phrase  is 
not  specified.  Based  on  whether  the  default  assumes  WRITE 
BEFORE  ADVANCING  or  WRITE  AFTER  ADVANC- 
ING inappropriate  spacing  or  overprinting  of  lines  can 
occur.  This  was  recognized  as  a shortcoming  of  the  language, 
and  subsequently  corrected  so  that  a default  is  now  specified. 

RESOURCES  REQUIRED 

An  accounting  summary  is  prepared  for  each  validation 
performed.  This  summary  includes  professional  personnel 
time  for  required  modifications  to  the  CCVS  (to  accommo- 
date any  compiler  peculiarities),  site  visit  for  acquisition 
of  raw  data,  evaluation  of  raw  data,  and  preparation  of  the 
VSR;  and  processor  time  required  for  executing  the  audit 
routines.  An  average  validation  lias  thus  far  required  73 
hours  of  professional  time  and  29  hours  of  clerical  time. 
The  processor  time  will  naturally  vary  with  the  computer 
used.  Execution  of  all  the  audit  routines  takes  approxi- 
mately 20  minutes  of  UNIVAC  1108  processor  time.  We 
estimate  the  average  cost  of  a validation  to  be  approxi- 
mately $1,000,  although  these  figures  are  subject  to  wide 
variations. 

CONCLUSIONS 

The  use  of  the  CCVS  in  validating  COBOL  compilers  at- 
tempts to  answer  three  major  questions: 

• Will  the  compiler  accept  the  syntax  as  defined  in  the 
COBOL  language  specification? 


• Will  till1  compiler  generate  the  appropriate-  object  code  to 
satisfy  the  semantic  requirements  of  the  language 
specifications? 

• What  unilateral  actions  does  the  compiler  take  when 
the  language  specification  leavi-s  the  result  up  to  the 
implementor? 

The  results  of  over  a year's  work  in  validating  a variety 
of  compilers  indicate  that  there  may  not  be  a compiler  which 
completely  conforms  to  the  COBOL  standard  In  a few 
cases  we  were  tempted  to  question  whether  the  compiler 
was  in  fact  compiling  COBOL,  or  some  other  language 
similar  to  COBOL! 

The  reasons  for  the  amount,  of  non-conformance  or  devia- 
tion from  the  language  specification  can  be  blamed  partly 
on  the  1968  COBOL  Standard.  Most  of  these  problem 
areas  have  been  resolved  during  the  development  of  tin- 
revision  to  X3. 23-1968.  As  a result,  we  feel  that  we  can 
expect  to  see  better  compilers  since  the  language  specifica- 
tions are  tighter  and  better  defined;  the  idea  of  providing 
standard  compilers  is  being  encouraged  in  the  marketplace 
by  the  users;  and,  most  importantly,  we  have-  a measure- 
ment, tool  which  can  be  used  to  determine  the  degree  to 
which  a compiler  conforms  to  the  Standard. 

We  recognize  that  the  Validation  System  is  necessarily 
incomplete.  But  we  also  are  convinced  of  the  importance 
of  having  some  capability  for  measuring  the  quality  of 
software.  What  we  have  learned  during  the  period  we  have 
been  validating  compilers  confirms  the  importance  of  soft- 
ware engineering,  and  thereby  the  importance  of  any  meas- 
urement tool  which  results  in  software  quality  improve- 
ment. 
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